Papers with adversarial annotator
RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced the capabilities in natural language processing. |
| Approach: | They propose a method to poison large language models by using annotators to rank a set of collected responses to generate longer tokens. |
| Outcome: | The proposed method can generate longer tokens without harming the original safety alignment performance. |